I made a buying robot that evaluates secondary market offers that has been running for over a year now. These days it is evaluating over 150k opportunities for buys each day. The chief problem is that of those 150k, only about 0.1% pass my criteria gauntlet and OF THOSE less than a dozen yield a SUCCESS_PENDING_SETTLEMENT. And OF THOSE only like 30% pass the "Cancelled, Payment Processing" check (I didn't put in a filter for payment date yet). That means of 150k evaluations in a day, 1 or 2 a day land as a success.
Most of these I can't control. I can't make people desire lower returns on their secondary market sells. I can't fix the payment processing checks on the LC backend. But I CAN control the gait of "list, evaluate, buy". The only signal that can be garnered here, though, is the ratio of SUCCESS_PENDING_SETTLEMENT to NOTE_NOT_AVAILABLE. This yields a huge blind spot into the real meaning of NOTE_NOT_AVAILABLE.
A naive interpretation would simply be "your buy was too slow." As a simple example, a List request takes a fraction of a second, and a buy request takes one or more seconds on a regular basis. This means that while a buy request is being processed by the LC backend, there are MULTIPLE other rounds of "lists" that other market buyers have gone through. Likewise, when I see the List come up, there may be multiple other people who have noticed the opportunity and are in the process of a buy request.
But I'm not fully convinced that is what is happening. The other explanation for what is happening is that I make a buy request, but the system that handles the buy requests simply hasn't received the information that the note has been put up for sale (aka, the buy is TOO fast).
I would HOPE that the note no longer being available would yield the "NOTE_DATA_STALE" state, but have no evidence on why that one shows up. Does anyone have some more insight into what the NOTE_NOT_AVAILABLE status means, or if there's a better way to distinguish between "buying too fast" and "buying too slow"?
You summarize the situation well.
I've had these same concerns for several years now. I don't get very good answers from LC. I suspect that the code for the secondary market gets very little maintenance, and the people who answer questions for customers are pretty far from knowing anything about how it works, so you get stock answers.
I agree that it is just astonishing what a large fraction of note purchase attempts result in NOTE_NOT_AVAILABLE. It happens so much that it seems surely something other than "some guy got there before you" is going on.
Because this market makes no transaction data available, there's no way to diagnose this from outside LC.
Playing in the LC secondary market has not been a worthwhile effort. The money I have invested is doing ok, but I can't get enough invested to be worthwhile. However, most of my costs are sunk costs (ie the effort to write and debug the software), and there is very little cost associated with letting it run, so I have let it run 24 hr/day, every day, since Oct 2016.
Strange, I've only seen the "Cancelled, Payment Processing" on my end, though I don't use the API. I have eliminated most of those by adding a "days since last payment" parameter of less than 30 to my filters.
I buy my notes manually from filters on NSR. Which is basically just API data with an interface.
I did have a few occasions where a note failed, but in the few cases (so far) it was because the price changed before I completed the order. I just look for the note again and usually find it with a lower price.
If I had to guess, "NOTE_NOT_AVAILABLE" probably means someone already bought the note (or added it to an order). "NOTE_DATA_STALE" probably means that the price has changed, so you need to find the new order_id.
Other than that, I would imagine that high-quality in-demand notes involve a bit of a speed game, just like on the primary platform. Might have to get like HFT traders and run your bots as geographically close to LC's servers as possible. (With the same ISP as well, preferably in the same colocation center)