From Newsgroup: comp.lang.python
Lawrence D'Oliveiro wrote:
Generally, when it comes to text editing, there is never a
description of "flowed text". Most of the editors do this.
+----------------------------+
| The quick brown fox jumped | <=== Each line is edited stand alone
+----------------------------+
| over the lazy hen, which |
+----------------------------+
| laid a scrambled egg. |
+----------------------------+
PDFMiner has heuristics to deal with this. I used it to build a post-processor for extracting customer and addressing information from invoices generated by a customer's accounting/reporting system, for automatically mailing them out. Once I had accumulated a suitable
collection of test cases for tuning the extraction parameters, it all
worked quite nicely. ??
<https://github.com/pdfminer/pdfminer.six>
Wow. This is EXCELLENT information for the group's tribal knowledge!
Since I've taken on the role for the past decades of testing every
suggested free PDF munging tool, I ran pdfminer on Paul's resume.
Not having heard of pdfminder.six before, I just now looked it up.
Apparently pdfminer.six is an open-source Python library for extracting
data from PDF documents because it understands the internal layout, coordinates, and typography of a PDF file.
How it works is, apparently, it first extracts every character or text
chunk along with its precise bounding box coordinates (x0, y0, x1, y1),
font family, size, and rotation angle.
Then it groups scattered characters into words, lines, paragraphs and text blocks. This is how it attempts to reconstruct the logical flow of text
from a format that natively has none.
In addition, apparently, it can extract images (in raw streams) and vector graphics data, though its primary superpower seems to be layout analysis.
It can tell us what is there in the PDF and where it sits.
To Paul's initial point about flowing text, PDF text is rendered by
throwing absolute coordinate instructions at the page (e.g., "draw the
letter 'Q' at X=100, Y=500"). There is no native metadata saying "this is a paragraph that should auto-wrap if the box shrinks".
To Paul's description of the illusion we can fall prey to, when a basic PDF editor lets us change text in a PDF, it's not re-flowing a paragraph. It's treating that line or text chunk as an isolated graphic box.
Hence, if you add a word using a basic PDF editor, there is usually overlap with adjacent text unless we manually fix the spacing issues it causes.
The golden rule applies to always go back to the source file, but if we're stuck fixing a finalized PDF, PDFminer.six helps with data extraction.
Let's install it into a VE sandbox, in case all hell breaks loose.
python -m venv venv
venv\Scripts\activate
pip install pdfminer.six
Collecting pdfminer.six
Downloading pdfminer_six-20260107-py3-none-any.whl.metadata (4.3 kB)
Collecting charset-normalizer>=2.0.0 (from pdfminer.six)
Downloading charset_normalizer-3.5.2-cp314-cp314-win_amd64.whl.metadata (47 kB)
Collecting cryptography>=36.0.0 (from pdfminer.six)
Downloading cryptography-50.0.2-cp311-abi3-win_amd64.whl.metadata (4.4 kB)
Collecting cffi>=2.0.0 (from cryptography>=36.0.0->pdfminer.six)
Downloading cffi-2.1.1-cp314-cp314-win_amd64.whl.metadata (2.6 kB)
Collecting pycparser (from cffi>=2.0.0->cryptography>=36.0.0->pdfminer.six)
Downloading pycparser-3.12-py3-none-any.whl.metadata (9.7 kB)
Downloading pdfminer_six-20260107-py3-none-any.whl (6.6 MB)
---------------------------------------- 6.6/6.6 MB 1.0 MB/s 0:00:06
Downloading charset_normalizer-3.5.2-cp314-cp314-win_amd64.whl (210 kB)
Downloading cryptography-50.0.2-cp311-abi3-win_amd64.whl (3.8 MB)
---------------------------------------- 3.8/3.8 MB 909.5 kB/s 0:00:04
Downloading cffi-2.1.1-cp314-cp314-win_amd64.whl (187 kB)
Downloading pycparser-3.12-py3-none-any.whl (62 kB)
Installing collected packages: pycparser, charset-normalizer, cffi, cryptography, pdfminer.six
Successfully installed cffi-2.1.1 charset-normalizer-3.5.2 cryptography-50.0.2 pdfminer.six-20260107 pycparser-3.12
[notice] A new release of pip is available: 25.3 -> 26.3
[notice] To update, run: python.exe -m pip install --upgrade pip
Now that it's installed, let's get out of the venv virtual environment
(venv) deactivate
Let's pick up a sample resume Paul kindly suggested in a prior post.
curl -o resume.pdf https://cdn.uconnectlabs.com/wp-content/uploads/sites/123/2025/05/MITCAPD-PhDI.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 112k 100 112k 0 0 291k 0 --:--:-- --:--:-- --:--:-- 293k
Now, to use the pdfminer.six stuff, let's start the virtual environment.
venv\Scripts\activate
Then extract that resume
(venv)C:\> pdf2txt.py resume.pdf > extracted_resume.txt
Drat! The result was an empty file. And it brought up Thonny too.
(venv)C:\> venv\Scripts\pdf2txt.py resume.pdf > extracted_resume.txt
Aurgh. Again, it brings up Thonny. It's the file associations.
(venv)C:\> python venv\Scripts\pdf2txt.py resume.pdf > extracted_resume.txt
OK. That worked. Now the PDF resume is a text resume, with bullets.
type extracted_resume.txt
--------------------- begin extracted resume ---------------------
Sample Resumes
PhD I Resume
Political Economy Ave., Cambridge, MA 02139
Phone: 617-xxx-xxxx * Email:
Upeg@mit.edu
JEAN UPEG
EDUCATION
Massachusetts Institute of Technology (MIT), Cambridge, MA
Candidate for PhD in Urban Political Economy and Governance
Dissertation: out of Control? Local Democracy Failure and Fiscal Control Boards
Princeton University, Princeton, NJ
B.S.E., Civil Engineering with Architecture, summa cum laude
EXPERIENCE
Fall 20XX
20XX
Community Innovators Lab, MIT, Cambridge, MA
Project Manager, "Innovation and Equity Transform America"; Research Assistant
* Authored federal taxation memo, coordinated authors, and wrote abstracts for memos to the Presidential Transition Team.
* Drafted grant proposals and policy memos. Participated in designing a model for equitable and comprehensive green retrofits.
2011-current
Currently collaborating with local and national labor and community groups on implementation.
Department of Urban Studies and Planning, MIT, Cambridge, MA
Teaching Assistant
* Conducted seminars, graded essays, and contributed to curriculum design. Classes taught totaled over 200 students and comprised
a doctoral research seminar, undergraduate policy course, and three masters planning courses. Conceived and taught graduate
mini-seminar.
20XX-20XX
Brookings Institution, Washington, DC
Brookings Research Fellow
* Awarded first pre-doctoral fellowship for dissertation research granted by the Metropolitan Policy Program.
* Created a dataset compiled from government sources on municipal finances and socioeconomics. Programmed rare-events
20XX-20XX
regressions to measure the impact of fiscal control boards in small cities. Performed qualitative case studies on the control boards
of Miami and Washington, DC through interviews with key actors, archival research, and evaluating financial reports.
* Presented at two national academic conferences for Political Science (7,200 attendees) and Planning (1,000 attendees).
P3 Planning Practice Project, MIT, Cambridge, MA
Research Assistant
* Researched four medium-size cities and their innovative community planning organization. Profiled planners of small cities using
20XX-20XX
national survey data. Created and maintained the project website.
Urban Institute, Urban-Brookings Tax Policy Center, Washington, DC
Research Associate II; Research Assistant
* Analyzed tax policy using statistical programs (SAS and Stata), with a focus on the distributional impact of national legislation, the
20XX-20XX
interaction of tax policies and valuation of fringe benefits, and state code relevant to low-income residents.
* Designed, launched, and maintained the Tax Policy Center website for press, policymakers, and researchers. Website received over
12,500 hits per day and was praised by Forbes, National Journal, and Business Week.
New York City Nonprofits Project, New York, NY
Research Assistant
* Developed a strategy to determine the economic impact of the non-profit sector on the city.
20XX-20XX
Professor Julian Wolpert, Princeton University, Princeton, NJ
Research Assistant
* Wrote a memo detailing the spillover effects of non-profits and value of non-profit tax exemption, focused on Philadelphia.
20XX
Fellowships and Awards
National Science Foundation Graduate Research Fellow, 3 years (2009-2012); MIT Presidential Graduate Fellow and Department
Fellowship, 3 years (2009-2012); civil and Environmental Engineering Book Award and David W. Carmichael Prize, Princeton (2006).
PROFESSIONAL AND PUBLIC SERVICE
Student representative, PhD Committee, Department of Urban Studies and Planning, MIT (2009-2011); Graduate Resident Tutor,
MIT (2010-2011); High school tutor, Maya Angelou Public Charter School, Washington, DC (2010-2011); Tax preparer for low income
households, Community Tax Aid (2008) and Lincoln Park Baptist Church (2008); Washington, DC.
PUBLICATIONS AND CONFERENCES
2 first author; 10 co-author; 2 conference presentations; 1 first author manuscript under review (refereed).
--------------------- end of resume ---------------------
Hmmm... I just realized, belatedly, that pdfminer.six was overkill for
*that* specific simple PDF file, but for a large complex multi-column
layout, I can see how Lawrence's suggested pdfminer.six can reconstruct
a logical reading order to give him/us a reliable way to pry data out.
--
Each Usenet post should strive to add useful value that wasn't there prior.
--- Synchronet 3.22a-Linux NewsLink 1.2