lucene入门-解析pdf(使用xpdf解析中文PDF详细过程)

deepfuture

下载xpdf和xpdf-chinese-simplified.tar.gz ，然后将xpdf-chinese-simplified.tar.gz解压到xpdf所在的目录形成一个子目录
http://www.foolabs.com/xpdf/download.html
The following packages are available:

• Arabic: xpdf-arabic.tar.gz (1058 bytes)
  
• Chinese/simplified: xpdf-chinese-simplified.tar.gz (835807 bytes)
  
• Chinese/traditional: xpdf-chinese-traditional.tar.gz (794568 bytes)
  
• Cyrillic: xpdf-cyrillic.tar.gz (1326 bytes)
  
• Greek: xpdf-greek.tar.gz (1164 bytes)
  
• Hebrew: xpdf-hebrew.tar.gz (1314 bytes)
  
• Japanese: xpdf-japanese.tar.gz (494624 bytes)
  
• Korean: xpdf-korean.tar.gz (470166 bytes)
  
• Latin2: xpdf-latin2.tar.gz (1435 bytes)
  
• Thai: xpdf-thai.tar.gz (1873 bytes)
  
• Turkish: xpdf-turkish.tar.gz (1140 bytes)
  

中文包的配置说明
Xpdf: Chinese Simplified support package
========================================
Xpdf project: http://www.foolabs.com/xpdf/
2004-jul-27
If this package includes CMap files, they contain their own copyright
notices and distribution conditions. All other files in the package
are Copyright 2002-2004 Glyph & Cog, LLC, and are licensed under the
GNU General Public License (GPL), version 2.
This package provides support files needed to use the Xpdf tools with
Chinese (Simplified) PDF files.
Contents:
- Adobe-GB1 character collection support
- ISO-2022-CN encoding
- EUC-CN encoding
- GBK encoding
Place all of these files in a directory, typically:
Unix - /usr/local/share/xpdf/chinese-simplified
Win32 - C:\Program Files\xpdf\chinese-simplified
Add the contents of the "add-to-xpdfrc" file to your system-wide
xpdfrc config file, which is typically:
Unix - /usr/local/etc/xpdfrc
Win32 - C:\Program Files\xpdf\xpdfrc
Alternatively, on Unix systems you can add these lines to your
personal xpdfrc file in $HOME/.xpdfrc.
能运行以下平台中
Precompiled binaries are available for the following machines:

• x86, Linux (staticly linked to Motif, t1lib, and FreeType):
  xpdf-3.02pl4-linux.tar.gz (11985186 bytes)
  
• SPARC, Solaris 10 (staticly linked to t1lib and FreeType):
  not currently available
  
• x64, Solaris 10 (staticly linked to t1lib and FreeType):
  not currently available
  
• x86, DOS/Win32 -- pdftops, pdftotext, pdfimages, pdfinfo, and pdffonts only:
  Win32 (built with MSVC): xpdf-3.02pl4-win32.zip (2046671e bytes)
  DOS6 (built with djgpp, with DPMI support from csdpmi5b): xpdf-3.02pl4-dos6.zip (1754621 bytes)
  

I've received reports of xpdf compiling successfully on the following systems (but binaries are not available on the net):

• x86 and MIPS, SINIX V5.4 (email f.miane@opengroup.org for binaries) (xpdf 0.5)
  
• Apollo 425e, DomainOS 10.4.1.2 (xpdf 0.5)
  
• m68k (HP-9000/425), HP-UX 9.0 (xpdf 0.5)
  
• Alpha, Linux (xpdf 0.7)
  
• POWER, AIX 4.2.1, gcc 2.8.1 (xpdf 0.7a)
  
• UltraSPARC 2, Linux 2.2.5 (xpdf 0.80)
  
• SPARC, Solaris 2.7, gcc 2.8.1 (xpdf 0.90)
  
• DG/UX (xpdf 0.90)
  
• LynxOS 2.5.1 (xpdf 0.90)
  
• HP-UX 10.20 and 11.00 (xpdf 0.90)
  
• MacOS X / Darwin (xpdf 0.92)
  
• QNX / X11 (xpdf 0.93)
  
• x86, OpenBSD 3.0 (xpdf 1.00)
  
• MacOS X / Darwin (xpdf 2.03)
  

,xpdf比pdfbox适应性更强，既能解析英文PDF，也能解析包括中文在内的PDF，但是XPDF实际上是在命令行运行
下面是在命令行运行，解析英文PDF后的效果
命令如下：
D:\workspace\testsearch2\xpdf>pdftotext ../htmls/xxxx.pdf xxxx.txt


 

编辑xpdfrc文件
cidToUnicode Adobe-GB1 D:\workspace\testsearch2\xpdf\xpdf-chinese-simplified\Adobe-GB1.cidToUnicode
unicodeMap ISO-2022-CN D:\workspace\testsearch2\xpdf\xpdf-chinese-simplified\ISO-2022-CN.unicodeMap
unicodeMap EUC-CN D:\workspace\testsearch2\xpdf\xpdf-chinese-simplified\EUC-CN.unicodeMap
unicodeMap GBK D:\workspace\testsearch2\xpdf\xpdf-chinese-simplified\GBK.unicodeMap
cMapDir Adobe-GB1 D:\workspace\testsearch2\xpdf\xpdf-chinese-simplified\CMap
toUnicodeDir D:\workspace\testsearch2\xpdf\xpdf-chinese-simplified\CMap
fontDir c:\windows\Fonts
displayCIDFontTT Adobe-GB1 c:\windows\fonts\SimHei.ttf
textEOL dos
在LINUX下可以查看add-to-xpdfrc文档，将该文档内容复制到xpdfrc中
解析中文PDF，需要加参数（同样的参数-enc GBK也能解析英文文档）
D:\workspace\testsearch2\xpdf>pdftotext -layout -enc GBK ..\htmls\readme.pdf
 
 

主要参数如下:
OPTIONS
Many of the following options can be set with configuration file com-
mands. These are listed in square brackets with the description of the
corresponding command line option.
-f number
Specifies the first page to convert.
-l number
Specifies the last page to convert.
-layout
Maintain (as best as possible) the original physical layout of
the text. The default is to 'undo' physical layout (columns,
hyphenation, etc.) and output the text in reading order.
-fixed number
Assume fixed-pitch (or tabular) text, with the specified charac-
ter width (in points). This forces physical layout mode.
-raw Keep the text in content stream order. This is a hack which
often "undoes" column formatting, etc. Use of raw mode is no
longer recommended.
-htmlmeta
Generate a simple HTML file, including the meta information.
This simply wraps the text in <pre> and </pre> and prepends the
meta headers.
-enc encoding-name
简体中文包只包含下面三种语言
ISO-2022-CN
EUC-CN
GBK

Sets the encoding to use for text output. The encoding-name
must be defined with the unicodeMap command (see xpdfrc(5)).
The encoding name is case-sensitive. This defaults to "Latin1"
(which is a built-in encoding). [config file: textEncoding]
-eol unix | dos | mac
Sets the end-of-line convention to use for text output. [config
file: textEOL]
-nopgbrk
Don't insert page breaks (form feed characters) between pages.
[config file: textPageBreaks]
-opw password
Specify the owner password for the PDF file. Providing this
will bypass all security restrictions.
-upw password
Specify the user password for the PDF file.
-q Don't print any messages or errors. [config file: errQuiet]
-cfg config-file
Read config-file in place of ~/.xpdfrc or the system-wide config
file.
-v Print copyright and version information.
-h Print usage information. (-help and --help are equivalent.)
下面我们使用JAVA将命令行包装起来形成一个类
package extract;
import java.io.*;
public class ExtractorCJKPDF {
/**
* @param args
*/
public static void pdf2text(String pdffile,String txtfile) throws IOException{

String pdfname=pdffile;
String txtname=txtfile;
String xpdfpath="D:/workspace/testsearch2/xpdf/";
String[] cmd=new String[]{xpdfpath+"pdftotext","-layout","-enc","GBK","-nopgbrk",pdfname,txtname};
//-layout表示保持原有的layout，enc指定字符集，-nopgbrk指定不分页
Process p=Runtime.getRuntime().exec(cmd);
}
public static void main(String[] args) {
// TODO Auto-generated method stub
try {
pdf2text("D:/workspace/testsearch2/htmls/123.pdf","D:/workspace/testsearch2/htmls/123.txt");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}