We've stopped using rSpec ...
Courtenay : November 3rd, 2008
...for new projects.
![]()
We upgraded the gems for one of our client projects, and the auto-loading / config.gems managed to completely break all our other projects, requiring upgrades, which caused weird breakages in weird places in some of the specs.
The app would refuse to deploy (rake tmp:create failed, because lib/tasks/rspec.rake was being loaded, and spec wasn't installed on the server). The annoying thing was that just having whatever.11 installed (I don't know the exact version) broke older apps on whatever.4 or whatever.0.2. .. so those had to be upgraded too. We wasted a day or two (three, maybe four developers) which equates to several thousand dollars in wasteage. It was also really infuriating -- the culmination of a few years of frustration of rSpec's weirdnesses.
After that, I found that some of the specs had never run (who knows why). It stopped reading spec.opts and started doing some weirdness with pending options. Finally, Rick just snapped, threw out rSpec and his Model Stubbing library, and now we're playing with a combination of rr, context, and matchy, trying to get a feel for a decent workflow again. It's sad and maybe a bit exciting to be on the edge.
What are you testing with?
56 Responses to “We've stopped using rSpec ...”
Sorry, comments are closed for this article.
November 4th, 2008 at 12:02 AM
We experimented with RSpec for a Rails project but the experiment never made it past that. We liked the BDD-style syntax and context, but found that it gave a false sense of security when it came to doing functional testing because the views were completely separated from the controllers under test. A well-reasoned philosophical difference, but it just didn't work for our team - bugs were exposed in the view code by the client and not our automated tests. Sure you can drop 'integrate_views' in your controller specs, but there's not a clean way to do it across all the specs in your project.
Since then, we've been using a combination of Mocha and Shoulda (we had been using Mocha even with RSpec). I've also been playing a bit with both matchy and context for personal Ruby projects - the RSpec-style syntax is what I really missed when jumping over to Shoulda.
November 4th, 2008 at 12:26 AM
I solved the view issue in my rspecon_railson_crack plugin (the tongue-in-cheek name highlighting the experimental nature of the plugin). Basically, I automatically integrate views for specs testing the output of controller actions, yet leave them stubbed by rspec for specs testing things like controller instance variables.
I understand the philosophy behind it, but we've run into the same problems.
November 4th, 2008 at 12:45 AM
Greetings, Shoulda.
It gives the behavior-oriented mentality, but with the normal basic underlying test framework.
That said, we don't use the Shoulda macros or macro creation methods anymore. They encourage some bad design approaches to testing, building too much magic into the tests. (should_be_restful being an example of the kind of thing that comes out of taking the macro approach too far.) We discovered that new rails developers thrust into the Shoulda world picked up (or perhaps were seduced by) those bad habits by following the design of the existing macros, unfortunately.
Still, I'm a strong proponent of Shoulda's more basic structures for testing. It's the simplest thing that can possibly work. :)
-- Morgan
November 4th, 2008 at 12:46 AM
Plain old Test::Unit, does everything you need without the magic. Just use modules for contexts and when unit testing, one class per method and one assertion per test case e.g.
I NEVER use fixtures, have a unittesthelper.rb for unit tests which requires mocha and a functionaltesthelper.rb for functionals which are the only tests allowed to hit the database (both these helpers require test_helper.rb containing the generics).
This works beautifully for me with no dependencies on other libraries.
Jamie
p.s. I never use any helper methods/setup/teardown's and instead just use a Factory meaning each test method has no dependencies on anything else at all.
November 4th, 2008 at 12:48 AM
Sorry, missed an 'end' for the ProductTests module.
November 4th, 2008 at 01:50 AM
Greetings,
I suppose all of it should be described. We use mainly boring old Test::Unit, with Shoulda (context/setup/should) encouraged for new tests, flexmock for mocking and fixtures for sample data.
I hadn't looked into 'context' before, but it looks a little like the parts of Shoulda we use. The main difference is the 'should' to define a test, which we've found puts developers into the right mindset to test the behavior, not the implementation. :) It's a little thing, and very 'soft', but it works.
We do explicitly test against our database at an accepted loss of performance, since we occasionally have to do complex queries, and ensuring their correctness is part of the tests.
We have a large code base and a wide range of Ruby and Rails developer skills, which guides our choices differently than many others.
-- Morgan
November 4th, 2008 at 01:53 AM
I'll give another vote for Shoulda. I checked it out initially based on the recommendation of Tobi over at Shopify.
It's done a pretty good job of helping stick to the behavior-oriented mindset.
November 4th, 2008 at 02:11 AM
+1 for Shoulda
November 4th, 2008 at 02:14 AM
I use shoulda+factory_girl.
The one thing I'd like to have is the blah.should syntax from the spec kind of side. So, I guess I'll have to try out matchy. And since rick ported his thingies to context_on_crack, the shoulda style matchers will be available for context. And, since we're changing everything, why not use rr, too. It seems like it will add synthesis style things to it too.
:)
November 4th, 2008 at 02:15 AM
At Reevoo we use Test::Unit for the most part, and mocha, naturally (generally bleeding edge seeing as the author is on our team) but we've tried to move away from too much stubbing/mocking and relying more on high-level functional tests.
I still like the idea behind RSpec but it hasn't gotten much momentum on our team; I have been playing around with Shoulda though and I think it works really well.
November 4th, 2008 at 02:31 AM
I like Rspec as it provides a complete BDD package. Giving you:
And with the rspec-rails plugin a bunch of useful helpers to make life testing rails apps easier (Though I think Shoulda's helper are more advanced here).
Regarding view specs, we don't use them at all! Instead we use: Cucumber. We have the customers write plain text features and we execute them as tests. This cuts through the application stack testing integration and output.
Getting customers/project managers writing tests is awesome :)
November 4th, 2008 at 03:33 AM
Hi, I'm Marius, and I'm an ex-Rspeccer.
Wow, that was a relief. I used RSpec on a customer project about 1 1/2 years ago and found it to cause way more trouble than gain. At that time I was on some soon-to-be-released Rails version, and found that it took so much maintenance just to be able to have my specs even run. I prefer to spend my time programming, not maintaining and patching the frameworks (bad OSS citizen, I know), and testing is my number one tool for programming.
Ever since, I've been feeling bad for not using RSpec anymore, thinking I might go back there one day. I really like the ambitious attitude of the RSpec team, and I'm sure it works well in encouraging programmers new to BDD/TDD to change their habits. For me, though, TDD does the trick and doesn't give me any overhead in maintaining the frameworkds.
I have been using Shoulda for the last half year or so, and really like the context/should syntax - which really is a tribute to the RSpec team.
November 4th, 2008 at 04:07 AM
Shoulda and Mocha.
November 4th, 2008 at 04:25 AM
Rspec encourages too much test complexity. Tests need to be simple in order to be effective. I used RSpec for about 6 months and then switched back to Test::Unit.
Mixing project specification, documentation and testing into one tool always seemed like a bad idea to me. I think embracing the constraints of a simple framework like Test::Unit is liberating because it allows you to focus on tests as code and not as some Frankenstein's monster amalgamation of specs, stories, runtime tests, and documentation all rolled into one.
November 4th, 2008 at 04:56 AM
+1 for Shoulda and Factory girl
Even though factory girl sometimes gets in the way, it is extremely handy, and a lot easier to work with, compared to fixtures.
Shoulda macros are ok, as long as you know WHY you need them :-)
November 4th, 2008 at 05:23 AM
This is the first time I've heard that anyone has had problems of this kind.
It should be trivial to fix by anyone who's familiar with the RSpec code - and that's a lot of people.
But we need to hear about the problems before we can fix them.
If you file a bug report at http://rspec.lighthouseapp.com/ with enough detail that we can reproduce it, I'm sure it will be fixed quickly.
November 4th, 2008 at 05:48 AM
I've been playing with test/spec, and more recently bacon. But I vendor them, so I get no surprises at release time.
November 4th, 2008 at 05:51 AM
I could never imagine throwing out our 2000+ examples on my current project.
November 4th, 2008 at 06:31 AM
FYI - for those of you experiencing the problem with the rake tasks, this was reported and fixed a week ago - just not released yet.
Bug report is here: http://rubyurl.com/TKOA
Fix is here: http://rubyurl.com/vfUo
November 4th, 2008 at 06:44 AM
Wahh wahh wahh.. rspec encourages you to write unicorn code, rspec put hot sauce in my eye, rspec was a bad husband, if it wasn't for rspec my code would have rocked, damn you rspecs! Err.. wait, rspec works for me. gems on the other hand..
November 4th, 2008 at 06:48 AM
I'm quite happy with rspec. The config.gems issue you mention has nothing to do with rspec IMO. It's a half-ass implementation in Rails.
I have yet to see where I've bundled rspec in a project and it has magically crossed the line and broken other projects completely unrelated.
It sucks you had these kinds of issues. Perhaps after a few more days of cooling off and clear thinking you'll be able to provide better information which may be able to lead fixing any issues in rspec that relate to this. Or if you discover they're not really rspec issues, but the way the projects were organized and relying on system gems, perhaps you can share that with the world as well.
November 4th, 2008 at 07:00 AM
When I TATFT, I use shoulda.
November 4th, 2008 at 07:03 AM
One question I have that stems form this post is based on the fact that you talk about "the culmination of a few years of frustration of rSpec's weirdnesses."
If you had that much invested in rspec, and you had frustration over the year's why didn't you not communicate with the community surrounding the tools you were using and invested so much time in? I just don't get it.
November 4th, 2008 at 07:15 AM
Just FYI, the context syntax is totally compatible with Shoulda's. You can use should to specify tests, etc.
I'm going to get a post up on my blog (as soon as my webby setup is complete...) that discusses why you should use context and such.
November 4th, 2008 at 07:25 AM
Hey Caboose. I suggest you read Aslak's comment above, as well as the thread on the rspec-users mailing list about your decision:
http://rubyforge.org/pipermail/rspec-users/2008-November/009873.html
Keep going with BDD and RSpec. They're a fantastic combination of methodology and testing, when done right. But you need to have a firm understanding of BDD for your specs to work well.
Also, come join us on the ML and bug tracker. We'd love to hear how you're using RSpec, as well as the problems you're encountering.
Cheers, Nick
November 4th, 2008 at 07:31 AM
Patrick,
Unit tests are not a silver bullet, they will never catch everything all the time. Relying solely on units to verify app correctness is insufficient.
I've had great success with several projects in the past decoupling controllers from views (even going so far as to test partials in isolation, and stub calls to partial from the view) because that isolation results in easier, more specific tests. That said, you do also definitely want something higher-level that runs through the whole stack, be it rspec stories/cucumber or selenium or whatnot.
Lately when developing I leave autospec running in the background for the units, and have a precommit hook to force me to also run our integration story specs before it'll let me commit.
November 4th, 2008 at 07:40 AM
Question: Where was your version control in all this? Was simply rolling back the offending project (and maybe the gems on that server) not an option? It sounds to me like you spotted a grease fire on your stove, and rather than put it out and examine the stovetop, you set fire to the rest of your kitchen so you could rebuild it.
What you describe is a problem with config.gems matched with the sort of dependency bug any other tool could have. That doesn't make it good, but I don't see how using three independent tools instead reduces your risk. Also, you don't claim here that RSpec was failing for you as a testing framework. How much more time are you going to lose getting back up to speed with another framework? Is it worth it?
November 4th, 2008 at 08:03 AM
I've found the rspec team to be extremely responsive to problem reports. The framework is very intuitive for me and has successfully transitioned me to a behavior driven development workflow.
November 4th, 2008 at 08:48 AM
I think the RSpec team has done a lot of good work. I'm currently using it on a project now. But I also use Test::Unit on some other projects and the one thing that always gets me is that it seems like I need to be too involved with RSpec vs. get in and get out with test unit.
When I first learned Ruby, moving from JUnit (I know, I know..) to Test::Unit was a no brainer. And honestly, I've never had many problems with fixtures in Rails, so I've missed a lot of that pain. These things worked for me because they followed known paradigms; junit/test unit, no problem, test database with yml data defining the models/tables, ok no problem.
I feel like I understand RSpec quite well, but it's just always felt like too much work. And I know a lot of people are fans of behavior and stubbing and mocking, but I only ever use those for cases that really need them - like external services or external dependencies. Mocking or stubbing models to me seems like, again, way more work than it's worth. That's what fixtures are for and I don't want to have to work around a bunch of things, I just want them to work right. Plenty of people will have issues with this opinion and that's fine.
My project that I'm working on now uses autotest, RSpec, mocking, stubbing, stories/story runner, etc, etc, etc. I've found I spent waaaaaay too much time dealing with just writing the tests. Way too much time. Not directing this specifically at RSpec, but I feel like in the Ruby community and maybe even in the testing community, BDD is better at creating more work for itself - I've never felt productive with BDD. Never.
Ultimately, with BDD it makes me feel like I spend way too much time coding tests and not enough time proving and moving on. To me, TDD and test unit are great because they're so simple and don't require all this discussion and abstractness. In the end, if I wanted to spend that much time screwing around describing things and stubbing or mocking example data or, frankly, just doing busy work, then I'd just go back to Java.
I wish the RSpec guys all the best in the future but I will stick with Test::Unit for my projects.
November 4th, 2008 at 09:51 AM
I've been using classic Test::Unit + fixtures for a year, and got bored by the slowness of tests + difficulty to maintain fixtures sometimes.
I've heard a couple of months ago about different framework, the cool kids were talking about rspec, shoulda, mocha, etc.
After some tests and evaluations, I must say I'm in love with shoulda and factory_girl. Those guys at thoughbot are doing a really great job, and I'm following all of their projects now. In a more personal way, rspec seems to have a syntax really similar to shoulda, but I really can't stand it, while I love the other one.
I'm very curious now about the next default choices for merb testing framework ;)
November 4th, 2008 at 10:14 AM
I like shoulda but still use rspec (though with mocha or rr for mocking), and am looking at bacon. I really don't like going back to assert_foo-land after a couple years in .should-land -- call me silly. We don't ever use fixtures (and delete them aggressively when we find them sneaking in), but recommend object_daddy or factory girl instead.
Rick
November 4th, 2008 at 10:15 AM
I use rspec, and am kinda curious about all of these horrible experiences everyone else is talking about... because it doesn't jive at all with my experience. We've got plenty of problems with our codebase and frameworks, but (unless I'm blind to it) rspec hasn't been one of them. What is everyone doing that makes rspec so frustrating to use?
Yeah, we did run into the problem where lib/tasks/rspec.rake required spec even though the gem hadn't been installed on the server. And seriously, the solution was trivial: rescue the failed require and conditionally skip the rest of the rake file. Ten minute fix. (IMHO, this is a failing in rails config.gem loading: no support for importing rake tasks from gems.)
But as Aslak and David posted above, rspec had an official fix for this problem already. And the fix might have been more widely publicized if people would complain in ways that are more useful.
November 4th, 2008 at 10:40 AM
Rspec/mocha/object_daddy
config.gem is a POS - as is gems in general. No surprise that there are problems surrounding it...
Just test.
November 4th, 2008 at 10:44 AM
After several years writing xUnit tests in .NET, I moved to Ruby quite recently, and one of the main attractions that drew me 'across the gap' was RSpec.
Seriously, if you're a Ruby programmer and you're not doing BDD, you're really missing out on something special that's right under your nose.
And if you're doing BDD in Ruby, I can't imagine a better choice of framework than RSpec.
The combination of Cucumber for acceptance testing and RSpec for unit testing is incredibly powerful. I've found the community around the project to be a credit to the software development community: knowledgeable, patient and helpful.
I think it's shame you guys have had such an extreme reaction to these problems. Can we tempt you back?
November 4th, 2008 at 10:50 AM
Funny..I just dumped rSpec on a project yesterday as well. And, it sounds like your last straw was similar to mine.
For some reason I couldn't vendor rSpec with config.gem, no matter what I did, it required the rSpec gem to be installed. I searched around for a fix, saw some discussion that'd it been fixed on edge, but I already wasted enough time and had been considering shoulda for some time.
I still use rSpec on a couple of gems and will probably continue to.
I just had enough of rSpec with Rails. I've spent too much time on tricky bugs that creep up at the worst times (usually when upgrading for another bug fix).
I'm giving shoulda and factory-girl a shot.
November 4th, 2008 at 12:31 PM
Thank god, I'm not the only one. :)
rspec just feels clumsy and unfinished. They're too detached from reality and I end up with a confused team writing specs that pass and code that fails. We're tossing rspec first chance we get.
November 4th, 2008 at 01:21 PM
RSpec was always a "nice in theory" thing for me. I've been happy with a mix of minitest and test/unit. RSpec just added too much overhead, and the way it modifies your classes on the fly never worked for me: the code you're testing should be the code you think you're testing.
November 4th, 2008 at 01:23 PM
A lot of this has to do with the fact that we all are programming on a platform that is a moving target. Good for innovation, but frustrating when things don't work or break just b/c you upgrade something. It's give and take.
Let me say this though, and I've said it time and time again in IRC, Rails auto-loading is the cause of a substantial amount of problems with Rails, gems, plugins, tests, etc... It was meant to be a convenience, a time saver so you didn't have to worry about putting in the "require" lines at the top of your files, but it ends up wasting more of your time debugging stupid shit.
Can we turn it off? Never even checked if that was a config option. If so, turn it off, and learn to use "require" like most other languages do :)
November 4th, 2008 at 01:37 PM
Do you think some of these problems came about because you were using pre 1 releases of RSpec ? As far as I understand anything before v1 is liable to change. That is why I did not change over when RSpec was all the rage. However I have started using it after the v1 release and I find it a good option.
Although I do think that it does consume allot of time and energy to write the specs, far more then I feel comfortable billing any clients I work for. Which is why I have not worked with RSpec on client projects.
I think I will give shoulda a go to see if it is easier and more streamlined option.
But if you are now going to move over to something which is again bleeding edge, you might come across similar problems in a year or so, when that technology matures.
Hamza
November 4th, 2008 at 03:21 PM
I guess this (matchy) is what you were talking about when you said it seemed everyone was making syntax-compatible rspec clones now.
I'm actually preferring bacon, which pollutes the namespace less by making you write things like
obj.should.be.nilinstead ofobj.should be_nilNovember 4th, 2008 at 05:11 PM
Can one of the folks that ran into problems using config.gem and rspec message me on github and/or lighthouse? (al2o3cr on both) I'm hoping that the bugs have been fixed in the 2.2rc release. If rspec is still having problems, I'd like to track them down before 2.2 goes final.
Thanks!
November 4th, 2008 at 09:12 PM
I'm interested in matchy, but "omg blog!! lol!!" seems to be down a lot?
Can we rely on matchy as a project?
November 5th, 2008 at 03:22 AM
I never really liked rspec. If your code is so hi-tech that you need to use multiple contexts and stuff, I like to make the code less hi-tech.
November 5th, 2008 at 04:36 AM
We have been using test/spec combined with test_spec_on_rails for the last year and it's really worked well for us. When we were deciding on whether or not to try a spec library we picked test/spec over rSpec primarily because we thought it was a questionable decision to include their own mocking library when mocha was already out there and stable. It also gave us the opertunity to continue using test/unit where necessary as we made the transition.
November 5th, 2008 at 05:19 AM
@jason That's a little silly. Your question really should be "Can we rely on Wordpress as a project?" My blog never went down before...it's just been down for a bout a week while I figure what I'm switching to from WP after an exploit let them inject JavaScript all over the place (including other static HTML sites on the same hosting account).
It'll be up before RubyConf.
November 5th, 2008 at 08:02 AM
Shoulda used Shoulda ;-)
November 5th, 2008 at 03:45 PM
I store my gems in my deployment and forget about it.
I like self containment. I would contain myself if it were possible.
November 5th, 2008 at 06:17 PM
Dealing with problems in third party code isn't a "waste of time." It's part of the job. My life as a developer got a lot easier once I learned to evaluate the pros and cons of things like RSpec on their merits, instead of "snapping" when things go wrong. Which they always do. It's the nature of the beast.
November 5th, 2008 at 10:49 PM
I'm pretty confused by this post ... I thought RSpec was pretty much the de-facto standard for testing Ruby. It certainly is with the Ruby coders I know and work with.
Personally, I adore RSpec and couldn't imagine TDD life without it. I've test-driven code with xUnit style libraries and the RSpec DSL is just ... beautiful
I like finding out about alternatives and about what people don't like about RSpec, but I'm not seeing a lot of what the problems actually are. The only problem I've ever had was a version of Autotest that was incompatible with a particular version of RSpec.
I've always wanted to use Bacon more, because it's *so* beautifully simple, but RSpec comes with more features and is more of a 'standard,' so I've stuck with it. Bacon+Mocha (or another mocking/stubbing library) could make for a yummy testing combination.
November 5th, 2008 at 10:56 PM
PS. I see some mentions of people using the rspec gem.
If you want to run into compatibility issues with your gems, by all means, don't freeze them! Sheesh - please freeze gems on production projects! I don't freeze gems for my personal projects, but I would never do that for say ... a production Rails app.
Also, it's conventional to stick rspec & rspec-rails into your vendor directory, where they should live forever. Once you've done that, use
rake specor./script/specorautotest- never use the rspec gem with a Rails project!I hope this post doesn't turn too many people off to RSpec - I adore it!
November 6th, 2008 at 04:27 AM
I'm a newbie to RoR and have been using rspec for the last few weeks for writing tests for the model layer. The syntax is good and i have had bot problems with it. The text mate plugin is also very useful
I am also looking at also getting my hands wet with cucumber in the very near future
November 7th, 2008 at 01:13 AM
I'm using the test environment that Rails provide me with. I love it. It's slim and simple.
For me rSpec and even Shoulda always seemed to be like the TV-show "pimp my car" where they take old beautiful and working cars and turn them into Frankenstein's monster.
November 7th, 2008 at 09:55 AM
I've tried Test Spec on Rails + Mocha for Rails projects. I'm using Bacon + Amok for non-Rails projects.
November 10th, 2008 at 08:35 AM
my only gripe is that rspec doesn't support 1.9--or does it?
November 12th, 2008 at 06:19 AM
roger - no, rspec is not yet 1.9 compatible - but that's going to be a high priority in December. If you'd like to contribute to making it happen, please search for tickets with tagged:"ruby19" at http://rspec.lighthouseapp.com/projects/5645-rspec and feel free to submit patches.
November 13th, 2008 at 03:47 PM
We use Shoulda and it affords us great tests without supplanting what is there with Ruby naturally.