In a way you are already moving forward with help from the community; that's what those pull requests are.
In addition to pull-requests, you have other tools at your fingertips
The project wiki
Which you may or may not have enabled. Like any wiki, this can double as a way for you to make announcements (publish guidelines for contributions, etc to your community) and a forum for discussion.
In addition to making you aware of changes others would like to see included in your project, these also act as a forum for code reviews. Others who are interested can see the changes and make comments directly against the commits included in the request.
This relieves you a little of the burden of reviewing incoming requests, though as the project owner you still have the final say on which to take and when they are taken.
Github also provides you with a rudimentary issue tracker for your project, allowing you to set release dates, assign them to individuals for resolution, classify them via tags, and so on .
When you get to know a few regular contributors well, you have the ability (under Collaborators on the project's settings page) to grant merge access to other people. This allows them to accept pull requests and push changes directly to the main repository.
Github provides you with everything you need to run a small development project and manage who can do what. It's up to you to make use of them.
If your project is successful enough to be attracting contributions from complete strangers, then it's off to a good start. But if you ignore your contributors one of them is going to fork it, accept the other people's changes, and effectively take the project away from you.