【环境部署】SPECTER模型-基于transformer的科学出版物
论文背景
标题：SPECTER: Document-level Representation Learning using Citation-informed Transformers
摘要：表示学习是自然语言处理系统的关键组成部分。像BERT这样的最新Transformer语言模型学习了强大的文本表示，但这些模型针对标记和句子级别的训练目标，并不利用相关性信息，这限制了它们在文档级表示方面的能力。对于科学文献的应用，如分类和推荐，嵌入提供了强大的终端任务性能。我们提出了SPECTER，一种基于预训练Transformer语言模型的科学文档的文档级嵌入生成方法，其依据一个强大的文档级相关性信号：引用图。与现有的预训练语言模型不同，SPECTER可以轻松地应用于下游应用，而无需特定任务的微调。此外，为了鼓励进一步的文档级模型研究，我们引入了SciDocs，一个新的评估基准，包括七个文档级任务，从引文预测到文档分类和推荐。我们展示了SPECTER在基准测试上优于各种竞争基线。
作者：【ArmanAI】Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld
CCF：B
会议：Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020
模型细节
安装Transformers
Python 3.6+, PyTorch 1.1.0+, TensorFlow 2.0+, and Flax.
通过Github仓库安装
1. 下载
## git clone git@github.com:allenai/specter.git
git clone https://github.com/allenai/specter.git
cd specter
## 通过浏览器下载archive.tar.gz更快
wget https://ai2-s2-research-public.s3-us-west-2.amazonaws.com/specter/archive.tar.gz
tar -xzvf archive.tar.gz
2. 安装环境
## 先安装conda，记住运行Anaconda.sh安装脚本时不要在root下运行，不然默认装到/root下，虽然在安装时选择其他路径可以让user使用，但是运行代码还是有可能出错，保险起见，就user下安装即可。
conda create --name specter python=3.7 setuptools
conda activate specter
# if you don't have gpus, remove cudatoolkit argument
#conda install pytorch cudatoolkit=10.1 -c pytorch
conda install pytorch cpuonly -c pytorch
## pip requirements.txt的命令最好转化成以下两条，用以在网络不好的环境下分别执行，防止一个执行了重复执行
## pip install dill jsonlines pandas scikit-learn
##https可以换成git试试
pip install -r requirements.txt
python setup.py install
3. 修改环境包依赖bug
运行代码报错，这是因为包的版本不对。
allennlp 0.9.0
overrides 7.3.1
其中overrides版本高了，应该将其改为低版本3.1.0
报错信息
(specter) user@ubuntu:~/model/model2/specter$ python scripts/embed.py \
> --ids data/sample.ids --metadata data/sample-metadata.json \
> --model ./model.tar.gz \
> --output-file output.jsonl \
> --vocab-dir data/vocab/ \
> --batch-size 16 \
> --cuda-device -1
Traceback (most recent call last):
File "specter/predict_command.py", line 14, in <module>
from allennlp.commands import ArgumentParserWithDefaults
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/commands/__init__.py", line 8, in <module>
from allennlp.commands.configure import Configure
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/commands/configure.py", line 26, in <module>
from allennlp.service.config_explorer import make_app
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/service/config_explorer.py", line 24, in <module>
from allennlp.common.configuration import configure, choices
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/common/configuration.py", line 17, in <module>
from allennlp.data.dataset_readers import DatasetReader
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/__init__.py", line 1, in <module>
from allennlp.data.dataset_readers.dataset_reader import DatasetReader
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/dataset_readers/__init__.py", line 10, in <module>
from allennlp.data.dataset_readers.ccgbank import CcgBankDatasetReader
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/dataset_readers/ccgbank.py", line 9, in <module>
from allennlp.data.dataset_readers.dataset_reader import DatasetReader
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/dataset_readers/dataset_reader.py", line 8, in <module>
from allennlp.data.instance import Instance
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/instance.py", line 3, in <module>
from allennlp.data.fields.field import DataArray, Field
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/fields/__init__.py", line 7, in <module>
from allennlp.data.fields.array_field import ArrayField
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 10, in <module>
class ArrayField(Field[numpy.ndarray]):
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 49, in ArrayField
@overrides
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/overrides.py", line 83, in overrides
return _overrides(method, check_signature, check_at_runtime)
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/overrides.py", line 170, in _overrides
_validate_method(method, super_class, check_signature)
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/overrides.py", line 189, in _validate_method
ensure_signature_is_compatible(super_method, method, is_static)
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/signature.py", line 102, in ensure_signature_is_compatible
ensure_return_type_compatibility(super_type_hints, sub_type_hints, method_name)
File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/signature.py", line 303, in ensure_return_type_compatibility
f"{method_name}: return type `{sub_return}` is not a `{super_return}`."
TypeError: ArrayField.empty_field: return type `None` is not a `<class 'allennlp.data.fields.field.Field'>`.
修改代码
pip install allennlp==3.1.0
4. 运行代码
此时再次运行以上代码，模型正常工作
脚本的输出大概是如下格式：