Twitter added extended tweet mode to their API and screwed up Autovist. Autovist helps you build a massive knowledge base by importing your tweets every night. When importing a tweet with a photo, Autovist has always been able to tweet it again and again looking exactly like the original tweet. It worked perfectly for years.
But then Twitter “enhanced” its API and returned different results by default. What was originally tweeted like this:
Winds up looking like this:
YUCK!!!! Big #FAIL. What genius at Twitter decided this was a good idea?
Lets dig deep into the guts of Autovist, reset the database a few days in order to walk through how Autovist actually captures new tweets, capture this one and look inside to see what’s going on.
The code below obtains all tweets since a given tweet id, which is a way of saying “this is the last tweet we have for the account, please send everything that has come after that.” This runs daily as a background job.
The code looks a bit complex, but all its really doing is gathering 200 tweets at a time and combining multiple arrays into a single array via flatten.
Next, we process all the tweets, discarding those that have been posted by Autovist (in order to keep only the new tweets):
Here, we’ve found the problematic tweet:
50: byebug 51: t = twitter_account.user.tweets.new 52: t.message = use_expanded_urls(tweet) => 53: t.message = t.message.gsub('&', 'and') 54: t.twitter_tweet_id = tweet.id 55: t.active = false 56: t.imported = true 57: t.import_uuid = import_uuid (byebug) t.message "Awesome Telegram bot searches our tweets + NVIC websites whenever u need the info. Try it out today! ☛… https://twitter.com/i/web/status/821478123422306304" (byebug)
So now the trick will be to figure out why the tweet text is coming back in this truncated form.
After digging around a bit, I discovered the changes Twitter made:
Wow! Twitter sure made tweets much more complex.
Now, according to the documentation,
all I need to do is add tweet_mode=extended to any endpoint. However, when I do that via the API wrapper…
…I receive a null text and full_text:
Autovist currently uses version 5.14 of the Twitter gem. I upgraded to 5.16, ran the tests and all passed. Good.
Rats! Same problem. OK. Time to look inside what is happening in the Twitter gem API wrapper itself.
Locate where the gem code physically resides via “gem which twitter”
I’ll skip all the byebug tracing it took to hone in on the problem and go right to the source: the Twitter gem’s Twitter class has, for years, defined a full_text method, which conflicts with the new values returned by Twitter’s API:
but now, as Twitter has announced:
The Twitter API dumped the text field and uses full_text instead. The Twitter gem API-wrapper defines a full_text virtual attribute that interferes with the API result. So that explains why we have lost #text and #full_text. Fortunately, we can still obtain the full_text via JSON:
(byebug) e.text <null> (byebug) e.full_text <null> (byebug) e.attrs[:full_text] "Federal vaccination recommendations are based on inadequate evidence and poorly executed studies https://t.co/yLPvWT3tnd #vaxfax"
I don’t yet know the full impact on Autovist; the next step is to re-record our VCR cassettes in order to have real API results to test against.
15 minutes later….
Woohoo! Re-recording the cassettes was no problem at all and there was only a single line of code that needed to be changed that brought us back to all tests passing!
After deploying the new version of Autovist, I’ll post some tweets with pictures and see if they import and post correctly from Autovist tomorrow…
A partial success: the tweets import correctly in extended format but when Autovist tweets them back out it receives a “Status is over 140 characters” error from Twitter because its not clear to us yet how to post them in extended mode. Ugh!!!
Posted question about how to use attachment_url and post an extended_tweet. Documentation seems to be lacking. I’m concerned about Twitter’s current ability to support developers because the announcement of attachment_url was made on Sept 19, 2016 and four months later promised documentation has not been delivered. (At least I got a few answers quickly.)
I learned that attachment_url cannot now be used to attach pictures;
Instead, you need to use media_id (returned for each uploaded media) for that; however, media_ids cannot be reused after a single use in a tweet! The YAML below is the internal representation of the tweet shown at the top of this article.
If you try to reuse that media_id in another tweet, you receive this error:
#<Twitter::Error::BadRequest: One or more media keys are not managed.>
This goes against the entire design intent of Autovist, which is to reuse valuable tweets over and over (including any attached media). Twitter is expanding what can be included in tweets, but its half-baked because its not possible to easily reuse extended tweets nor to re-create them using the API.
Ideally, media should be able to be included and reused via a simple link to one or more images previously uploaded to Twitter. Forcing a one-upload/one-use policy is antiquated in a world of 12-factor/cloud apps.
The way Autovist import has worked for the past 3 years has been to use :full_text in the statuses/update endpoint call. It still works perfectly – but only for “classic tweets” that are strictly limited to 140 chars including media.
Latest posts by Chris Downey (see all)
- Dramatic improvement by replacing ActiveRecord loop with single SQL query - March 17, 2017
- Activist best practice: auto-archiving web pages - February 4, 2017
- Pre-calculating vaccine ingredients: from PostgreSQL to HTML data-* - February 1, 2017