Debated posting this, but so many ramifications I felt it best to do so. Original post in my forum. Reported here for broad applicability. The question has been why my systems (and I imagine those of many) have been less capable of doing business since the latest LC change. I detail the change and consequences below.
The issue is that P2P-Picks relies on LC providing its loan application data via the CSV or API, and that these are no longer the most timely way to get loan application data. It is affecting me personally and all of my clients, too, and there isn't much I can do about it except complain to LC, which I have done.
It used to be that when you clicked the link to get the CSV, LC would instruct its systems to generate a fresh CSV based on the loan applications that were available at that very moment. This, I imagine, was expensive for them when thousands of people were clicking at the magic hours. Hence why we saw rate limiters in place last November. In the last few days, they moved to a model of linking to a static CSV available for download that is updated for freshness just once every minute. While this relieves the stress off their servers, it has had some unintended (or perhaps intended) consequences.
Here is the rub. LC releases the loans onto their platform around 30-40 second after the hour. The loans are available to website users a full 20-30 seconds before they ever even appear in the CSV. The API has always been out of date a number of seconds, just like the CSV is now. Now, the absolute fastest way to get loan application data is to use the website interactively via their filter processes (or to screen scrape for automation). This puts all of us who want to do active loan selection in a position of either sitting at our terminals every release twitching like some kid playing a video game or breaking the rules (even more than we already have to do) and resorting to screen scraping loan data off the website in order to automate things. Of course, the web pages don't even have a majority of the credit variables anyway, so this may be of limited value.
I have witnessed this occur on both the fractional and whole loan side. Reliably, at 30 after the hour, new loans will appear. Within a few seconds, the "good stuff" (to the extent you wish to believe in such a concept) is either purchased outright (in the whole loan market) or is mostly already full (on the fractional side) by the time the other two loan data retrieval methods are updated. Here is an example from 6pm a couple nights ago to illustrate:
5:55:00 - Initial count: 31 whole loans on platform
6:00:05 - Unused whole loans expire (12 hour rule): 19 whole loans on the platform
6:00:30 - 170 new whole loans added to platform (as viewed from the webpage) - total inventory = 189
6:00:45 - 24 whole loans disappear off the platform - total inventory = 165
6:01:00 - browseNotes.csv and LoansBrowseLoans update with just 165 whole loans in them
Why would LC do this? I don't want to go into pet theories, even though I have some, before allowing LC a chance to respond. And to be clear LC is absolutely still providing an even playing field. Same rules for everyone. It's just that this change has benefitted a certain type of user (the twitching kid or the scraper).
My recommendation is to synch the timing of all forms of loan application data, website, CSV, and API. If the Royal Bank of Canada wrote Thor to synchronize orders across multiple stock exchanges, I'm pretty sure this is a feasible request.
What happened to LC's own autoinvest? Does it get loans as fast as screen scrapping?
That is a very important question. All third party automated investment systems designed for retail investors that I know of rely on the now-outdated CSV and API for loan input data. Either there are a lot of twitch buyers sitting on the site, or LC's own Prime service is getting first crack at automated investing. Those are the only two options for explaining why so many fractional loans are largely filled by the time of the CSV/API update. If it's the latter, it could be argued that their internal systems have an unfair advantage, as they wouldn't have to resort to screen scraping to organize the data (which takes time to poll individual web pages).
This is troublesome. I don't understand the need for an API if the information is stale. I can understand limiting the CSV file and metering to deter screen scraping. However, an API should provide the same level of service as their front end. This may have the opposite affect of now requiring auto invest users to use their front end (screen scrape) to place orders.
PLS does have the feature of using the web site front end to place orders based on previously saved LC filters. However, there is no way to apply sophisticated logic such as AI, ratios,etc.. In other words, we can still place orders faster than those who are doing it the manual way. However, we won't have the ability to analyze the loan data until the API or CSV is updated. IMHO, the reason for the API is the ability to automate the loan selection process. I don't understand why they would give the web site an unfair advantage? I see this as a migration from the API to the web site which may be an unanticipated consequence.
Bryce, in your analysis, is the API updated at the same time as the CSV? Are there any advantages with one over the other?
The API appears to update simultaneously with the CSV, as least when it matters at 1 minute after the hour. Historically the API was a little stale in my experience. This is all just my limited experience. I'm not going about this in a super scientific way, but have observed a number of times.
Timing seems to matter! I was quite happy with P2P-Picks and BV the way it was working! I suspect LendingRobot is affected as well. We have another account using it but it is pretty selective so not sure if the note investing have slowed down yet. They seem to be.
Title: Mad Rush at Lending Club Loan Release Time: Part I
Posted by Anil Gupta | 08 Jun 2014 22:11:48 | Category: Lending Club
https://www.peercube.com/blog/post/21
I encourage others to verify my own observations.
Another issue is when to download the CSV or API data? Prior, the note count could be used to determine when to download the CSV/API loan data. As soon as the note count increased by a significant amount, you could download the loan data in one shot. Now, I can't think of a way to determine when to download the API/CSV. There is no note count in the API (which is absolutely ridiculous). The only thing I can think of is to continue to monitor the note count then continuously poll the CSV/API for an update? If that is the case, then we are back to the main problem of the expense/load to handle the poll traffic. That is, we have to resort to a continuous download of the CSV/API until it catches up with the note count. Any ideas on how to make this a smoother process?
I can't even begin to imagine why the website, API, and CSV can't be updated simultaneously ...
LC has set up system that encourages screen scrapping and exploiting the shopping cart lock out while the crafty users wait for the CSV to update. I mean, if I wanted the best loans, and was willing to break LC's rules and possibly the law (is willfully delaying an open market order, and preventing others from placing an order, while one waits for updated information to be posted, illegal market manipulation? -- well, we know people are doing it already), I would design my automation to screen scrape my prospects from the website based on the limited info posted there, put them in the shopping cart, then wait for the expanded CSV to be posted before I select which ones to keep....
This can't be the vision that LC has in mind for the future of Automation ....
I use the shopping cart for recent listings of filters-applied EFG loans because I have more uninvested cash than the new listings will consume. If LC had a way to sort KEEP from DISCARD into two piles I could get through the list faster. I don't waste time "waiting" for a .csv to update. I go straight to work on what's there and check the 3rd sites when I think they've had time to publish. Waiting instead of loading the cart would mean losing all the lower amount, 36 month loans without even being able to see them in order to make a decision! HOW ON EARTH IS THIS CHEATING?? It seems like common sense to me.
If LC's filters had the same filter options the 3rd party sites do there would be a chance I could use their automation.
A point of interest to me in all this recent discussion about the timing of available loans is that Interest Radar is very, very fast to have them all. Rev must have a secret sauce. Unfortunately for me, I'm addicted to an additional data point only available on a different service, but it too is ready while I'm still paring down my list.
I don't know if that last post was directed at me, or not (I never called anything "hoarding" or "cheating", nor did anyone else), but I completely agree that as long as LC is going to tilt the field to encourage screen scrapping and use of the shopping cart to retain notes while you think about whether or not you really want to buy it, well, the crafty investors will do just that. In fact, I posted on the PLS section (
http://www.lendacademy.com/forum/index.php?topic=1907.0) several months ago that I'd love to see PLS modified to do a quick LC filter grab of notes, then exploit the shopping cart lock out while it runs it's models.
As you pointed out, it works best if you have lots of cash to invest, and grab all or nearly all the new notes, which is why I think this has probably become the choice tactic of the institutional investors.
SO, I'm not being judgmental, I'm right there with you, that it is the best way to get the best notes. When I used to "hand invest", I'd do the same thing. Refresh like a mo-fo until the new loans appeared, quick filter, sort, click-click-click the top notes with the highest yields, add to cart .... then start looking at the quality of what I selected. However, that strategy stopped working about 6-7 months ago when I found I could not do it by hand fast enough to get the top notes in my filter, which led me to believe the game had changed and the computers were now doing what I was doing, but doing it much faster.
But, while I do admit to doing this myself, I've always wondered if the tactic was legal, in that I was preventing others from participating in a fast paced market while I took my sweet time to decide what I wanted to keep, and what I wanted to throw back. Can you imagine if the SEC discovered that some hedgefund had discovered a way to prevent other's from trading a stock while they ran models to see if they wanted to buy it or not .... I'd imagine they would not be amused. That's a much closer analogy to consider than taking two pairs of pants into the dressing room.
I see people locking up loans from high income borrowers right at feeding time, but then release them after further review. There's definitely a lot of gaming the system going on to get some perceived "edge." Someone needs to write a version of "Flash Boys" as it pertains to the Lending Club world.
It's not cheating to use the website to buy a bunch of loans right at release. Nobody said it was. To the contrary my original post said the playing field is indeed even. Everyone gets the same info at the same time. We are just complaining that the supported official ways to get loan applications are unequal. LendingClub is my main hobby and I have spent hundreds of hours building models, culturing relationships, and writing code to automate loan selection that is based on the premise that use of the API or CSV will allow me to be competitive in the marketplace. For two years that was true. Last week it got turned on its head. We simply can't do the type of analysis we want to do now.
It is not too difficult for LC to implement a "shopping cart" like that of Amazon:
http://www.amazon.com/gp/help/customer/display.html?nodeId=468468
Curious to hear LC's reply. You sent this to them Bryce?
I'm also curious to hear LC's reply to Bryce.
Fairness is quite the sticky wicket. API, CSV, and the site all feeding at the same time seems fairest to me.
GS, sorry about the pants analogy. That was bad. I like your stock analogy much better. I'll edit mine out.
Fred, the Amazon system would create a lot of one-note orders from individuals. It is fair, but the order proliferation could be a problem if enough manual selectors hang on.
Bryce and GS, I don't remember who used the words hoarding and cheating in a different thread yesterday, but it wasn't you and that viewpoint is not new. It has long been a criticism of the system and has sometimes been directed at individuals (not at me, but feel free to do so now!).
btw, IR was slow this morning and too late for some loans. I'm grateful that the 3rd sites are still operating at all and for all the work their developers have done. Don't know why LC ignores the main reasons people are using them while trying to be so "improved".
I'm sure the third party sites appreciate the acknowledgement of value added, but let's keep it on topic. It is more likely that this was a change that wasn't so well thought out and had unintended consequences.
I sent the observations to LC (one of the gentlemen in the investor services dept.) and haven't yet heard a reply beyond acknowledgement of receipt and forwarding. But, it is the weekend. Hopefully we can see equal treatment across all info feeds in the near future.
Bryce,
Do you have any feel for how many of the loans that seem to be getting scooped up before you can get the data to run your analysis would have been top P2P-Picks? Is it possible for you to analyze the loans that are selling quickly to determine any pattern of what might be making them desirable?
I think the concept of "feeding times" (currently, 4 daily) should now be abandoned.
None of this asymmetric timing issues would matter. It should also solve a lot of bursty demand of CPU, memory, network, etc.
On the whole loan side, my picks went from many per release to zero or one, so it was a great impact. On my personal account and the picks list, the volume declined precipitously. I have over $1100 in my personal account which has not happened in years.
I'd be willing to bet most people are just trying to gobble up high interest rate notes indiscriminately.
A fundamental question is whether a note locked up in a shopping cart (which is fully funded) will even appear in a CSV/API call. You need that to be true in order to get the full credit variables for a future analysis.
Looks like the CSV delay is permanent (from Lending Club support):
Thank you for your email and interest in Lending Club. If you are using an API that gets loan information from the "Download All" in our Browse Notes tab, there is a small delay. The "Download All" information is updated once per minute rather than continuously. This is a permanent feature to allow equal access to the platform for all investors.
My question also asked about the API. However, the response appears to be referencing the web site as the API (which doesn't make sense). I replied with clarification regarding the API and will repost. However, my own analysis confirms that the API is indeed delayed (I believe in sync with the CSV).
I am trying to stay open minded about this whole thing. However, I just don't see this as a viable solution. Although the rules are the same for everyone, this is definitely not "equal" access to all investors. If you are one type of investor, you get the front of the bus. If you are another type, you get the back of the bus. Just change your investor type and you get to the front of the bus. Therefore, I believe most savvy investors are going to have to change their investor type, and now we are all competing in much the same way we did before things changed. Thus cutting out the so called "equal" access. In addition, you now have to use the limited filtering LC provides or use the cart hole.
Ugghhh. I wonder if it's time to make the switch to Prosper?
Maybe off tangent but applicable:
Even if the API isn't delayed by content(which I believe it is), it is appears to be delayed in execution. On average, the download time of the CSV file takes me about 4-6 seconds without throttle on repeat downloads. Downloading the notes via API, appears to be about 5-10 seconds on first attempt. Then up to a minute on repeat attempts. Since there is no indicator on when the API is updated, you are forced to repeatedly download the CSV file. Hence, migration from the API.
I declare the API/CSV dead. Well, the API was always dead. But now it's like a zombie, infecting the CSV with its malaise. Dead and greyed, patchy flesh hands grasping at the last wisps of higher-interest borrowers as they run away from the pack, just out of reach.
Time to adjust.
Fred, that's where I said I wasn't going to go in my original post. But that's my thought, too. Only one way to tell...find a few people with PRIME, identify the loans bought into, and compare with lists of loans that never made it to my archived CSVs. I really hope that isn't happening, because it would be an implicit acknowledgement that there's a "best" set of loans, something they've vociferously argued against the entire time.
Bryce,
I am testing the "Automated Investing" service with them right now.
How can I help with this?
Ray
I sent LC a note tonite telling them what I think about the recent API/CSV/Web changes.
Titled "Lending Club ecosystem, battles with your customers, and the API"
I've attached a copy which is identical, except that I redacted my name.
Hope you guys back me up on this. If you agree, send them a note telling them so. If you think I'm full of crap, send them a note telling them so.
I sent it to my normal contact, and also Renaud Lapanche (CEO).
I'm a retail investor who has never used API or a 3rd party service, and admittedly I know nothing about the IT infrastructure of Lending Club or Prosper. But my dollar is just as valuable as anyone else's, so here's my opinion. From a big picture perspective, any move that puts individual investors (not concerned about institutions for this point) on a common playing field is a good move in my opinion. The API and 3rd party services, by using advanced speeds and data, have frequently put the common investor like me at a disadvantage. I shouldn't need to know how to code to invest. I shouldn't need to sign up for a 3rd party service to gain greater speeds to invest. I should be able to log onto LC or Prosper directly, or use their automated investing tools, and put money to work myself. And more and more, that's what I can do. Personally I prefer Prosper and I've been finding many loans that meet my criteria each day. I don't expect them to cater to me, and I'm not sure why anyone else expects to be catered to. No 3rd party service is a part of Lending Club, so I wouldn't think they have any obligation here.
Just speculating, but I believe the point LC is attempting to make is that they want you to invest through their Automated Prime service or don't bother. In this way they enable you to pick the note allocation, or rate of return you are looking to achieve, while simultaneously killing off the entire "High Frequency Front-running" (so to speak) created by all the programs using the API.
With the overwhelming majority of investors, or more importantly the minority of investors with the majority of the money, using the Automated Investing Service they can foresee the upcoming Demand and try to match it up with the Supply of new notes in an orderly manner. Although this may crimp the style of the few who have been able to cherry pick notes to-date, in the long run the new approach seems "fairer" to the "Independent Observer" and is beneficial to lending Club from an operational standpoint.
Although this may upset the forum readers , it is exactly what I would if I was running LC.
If you read LC Automated Investing information, they are ranking investors internally. The more cash that you have the better your access to loans:
"The frequency of orders is based on the cash balance of your account" -- sounds like to me they are going to rank the accounts from highest cash to lowest. IMO, someone that is a retail investor like me will always be at the bottom of the list as I try and keep my cash invested.
As written in an earlier post, cutoff the third party tools and I will be leaving LC too.
"Your Automated Investing account will be reviewed automatically up to four times per day to determine the level of available cash in your account, overall demand on the platform, and availability of Notes matching your investment criteria.
Automated Investing prioritizes accounts with more available cash as a percent of the overall account size and places orders for those accounts first. Automated Investing does NOT prioritize accounts based on overall account size or investment criteria." (
http://kb.lendingclub.com/investor/articles/Investor/How-does-PRIME-determine-when-to-make-an-order-for-my-account )
I love when you spin a document anyway you want, but if you bothered to read the entire paragraph you will see you are totally wrong.
The obvious problem is that as soon as the asymmetry goes away, manual buyers suddenly get fucked as hard as the API buyers are today, but worse and in reverse. Today, a person frantically f5'ing the note listings can nab a good deal of high-desirability notes (that are funded in <5 seconds of listing) and filter through them at their leisure. Tomorrow, this person will be 100ms behind a computer controlled program and will get nothing.
I think it's faulty thinking that investors _want_ to interact with the website in the long run. It's so tedious to log in 4 times a day and mash your F5 button. I've had to interrupt my day countless times to respond to my LC alarm. There is a non-trivial group of people who want automation but not LC's own automation. Right now that ability is totally broken and I'm going to need to start making withdrawal as cash piles up.
Are we sure it's manual investors that are snatching up the new loans in 5 seconds, or could it be LC's Auto Investment users have an edge over everyone?
Certainly some of the loans I would buy are gone by the time the API sees them (they are 95% full according to the API download and by the time my order goes in it misses) or occasionally the API doesn't even see a particularly good one. Recent testing shows this isn't overwhelmingly the case.
Other reasons for the lower volume is that nothing is meeting my expectations (perhaps LC is setting rates closer to what my model suggests) or the overall type of loan (read quality according to my expectations) isn't available right now.
For what it's worth I was able to invest in 57 notes (p2p picks top 25% CDE) between 2 release times today. I was a bit surprised to see them this high, but perhaps others are using top 10%?
Without a doubt it has gotten a lot better last week or so, hopefully this will continue at least till the IPO:)
If they aren't going to give back the not issued note money soon the inventory will grow even faster. This is the first time I've been out of investment money (intentionally), so this is a new complaint for me. I have $5, 275 of "Committed Cash", which I think is a personal record.
Lending Club's results happen when notes are issued, not when our cash is committed.
If people are seeing committed cash build up, then I would take that as a sign that LC has already issued their targeted amount of loans for the quarter and they are holding back issuances to give next quarter a jump start.
If loans issuances continue normally and committed cash holds steady or drops through the end of the month, then I would see that as a sign that LC is having trouble meeting their numbers and wants to capture as much business in the quarter as possible.
Last quarter the slowdown in issuances started around the 24th of the month.
Sent from my iPhone using Tapatalk
There's a post in the Lending Club borrower's forum where a borrower states they were fully funded and approved on 9/25 but told they will get their funds on 10/2. More evidence that LC is holding business till next quarter...they must have this quarter's numbers in the bag already.
Sent from my iPhone using Tapatalk
I think we have wondered off topic here.