Ingesting PDFs using OpenAI GPT-4 Vision (2024)

PDFs are everywhere, but getting information out of them can be tough, especially when they're packed with charts and tables. That's why I started using OpenAI's GPT-4 Vision to make things easier by converting PDFs into Markdown, a format that's much simpler for computers to read.

Traditional tools for pulling text from PDFs are hit or miss. They might miss important details, especially if the PDF has lots of visuals. This inconsistency is a big problem when you're trying to understand or use the information in those PDFs.

Markdown is great for this because it's straightforward and structured, making it easy for AI to understand. OpenAI even uses Markdown to talk to ChatGPT, which shows how useful it is.

Here's what I did: First, I turned each page of the PDF into an image. This way, I didn't lose anything, like charts or images, that I might miss if I just tried to pull out the text. Then, I used GPT-4 Vision to read those images and turn them into Markdown text. GPT-4 Vision is smart enough to handle complex layouts and visuals, so I ended up with Markdown that kept the original PDF's content and structure.

I wrapped all this up into a few Python scripts to automate the process. There's one script to turn the PDF into images, another to convert those images to Markdown with GPT-4 Vision, and a third to clean up the Markdown and get rid of anything we don't need, like placeholder images or page numbers. There's even an optional script that puts all the cleaned-up Markdown into one document.

This method isn't perfect, but it's a big step forward in making PDFs more accessible and easier to work with. Manually converting PDFs to Markdown by hand isn't realistic on a large scale, and just pulling out the text and chopping it up into chunks isn't enough, especially if you're missing out on important visual information.

Check out the GitHub repo for the scripts I mentioned. I hope this method helps you see the potential of AI in making it easier to work with PDFs and other documents. Feel free to reach out to me on LinkedIn if you have questions or want to chat about it.