If you can make it on a weekday/afternoon, you will be the only person there. At least for the next few weeks until summer. I'd be willing to join sometime next week.
If you go on a weekend, it will suck.
I have ridden my dirt bike at Brown Mountain numerous times and would love to take my Samurai down some of the dirt bike trails. Maybe we can get a weekday trip organized with a group. It definitely won't be for those worried about scratches and dents.