ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] #6391

paolovic · 2024-12-19T09:40:51Z

Reminder

I have read the README and searched the existing issues.

System Info

$ llamafactory-cli env

llamafactory version: 0.9.2.dev0
Platform: Linux-4.18.0-553.27.1.el8_10.x86_64-x86_64-with-glibc2.28
Python version: 3.11.10
PyTorch version: 2.5.1+cu124 (GPU)
Transformers version: 4.46.1
Datasets version: 3.1.0
Accelerate version: 1.0.1
PEFT version: 0.12.0
TRL version: 0.9.6
GPU type: NVIDIA L40S-48C
Bitsandbytes version: 0.45.0

Reproduction

Dear Llama-Factory team,

I created a a LoRA adapter which worked out of the box for Llama 3.3, thank you very much.

Unfortunately, when I wanted to evaluate the model with the adapter, I get this error

llamafactory-cli train examples/extras/nlg_eval/llama3_lora_predict.yaml

***** Running Prediction *****
[INFO|trainer.py:4119] 2024-12-19 10:32:25,907 >>   Num examples = 347
[INFO|trainer.py:4122] 2024-12-19 10:32:25,907 >>   Batch size = 1
[rank0]: Traceback (most recent call last):
[rank0]:   File "/packages/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>
[rank0]:     launch()
[rank0]:   File "/packages/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]:     run_exp()
[rank0]:   File "/packages/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp
[rank0]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]:   File "/packages/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 128, in run_sft
[rank0]:     predict_results = trainer.predict(dataset_module["eval_dataset"], metric_key_prefix="predict", **gen_kwargs)
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer_seq2seq.py", line 259, in predict
[rank0]:     return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer.py", line 4042, in predict
[rank0]:     output = eval_loop(
[rank0]:              ^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer.py", line 4158, in evaluation_loop
[rank0]:     losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
[rank0]:                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/packages/LLaMA-Factory/src/llamafactory/train/sft/trainer.py", line 126, in prediction_step
[rank0]:     loss, generated_tokens, _ = super().prediction_step(  # ignore the returned labels (may be truncated)
[rank0]:                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer_seq2seq.py", line 331, in prediction_step
[rank0]:     generated_tokens = self.model.generate(**generation_inputs, **gen_kwargs)
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/peft/peft_model.py", line 1638, in generate
[rank0]:     outputs = self.base_model.generate(*args, **kwargs)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/generation/utils.py", line 1972, in generate
[rank0]:     self._validate_model_kwargs(model_kwargs.copy())
[rank0]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/generation/utils.py", line 1360, in _validate_model_kwargs
[rank0]:     raise ValueError(
[rank0]: ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] (note: typos in the generate arguments will also show up in this list)
[rank1]: Traceback (most recent call last):
[rank1]:   File "/packages/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>
[rank1]:     launch()
[rank1]:   File "/packages/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank1]:     run_exp()
[rank1]:   File "/packages/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp
[rank1]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank1]:   File "/packages/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 128, in run_sft
[rank1]:     predict_results = trainer.predict(dataset_module["eval_dataset"], metric_key_prefix="predict", **gen_kwargs)
[rank1]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer_seq2seq.py", line 259, in predict
[rank1]:     return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer.py", line 4042, in predict
[rank1]:     output = eval_loop(
[rank1]:              ^^^^^^^^^^
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer.py", line 4158, in evaluation_loop
[rank1]:     losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
[rank1]:                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/packages/LLaMA-Factory/src/llamafactory/train/sft/trainer.py", line 126, in prediction_step
[rank1]:     loss, generated_tokens, _ = super().prediction_step(  # ignore the returned labels (may be truncated)
[rank1]:                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/trainer_seq2seq.py", line 331, in prediction_step
[rank1]:     generated_tokens = self.model.generate(**generation_inputs, **gen_kwargs)
[rank1]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/peft/peft_model.py", line 1638, in generate
[rank1]:     outputs = self.base_model.generate(*args, **kwargs)
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/generation/utils.py", line 1972, in generate
[rank1]:     self._validate_model_kwargs(model_kwargs.copy())
[rank1]:   File "/environments/llama_factory_uat/lib64/python3.11/site-packages/transformers/generation/utils.py", line 1360, in _validate_model_kwargs
[rank1]:     raise ValueError(
[rank1]: ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] (note: typos in the generate arguments will also show up in this list)
[rank0]:[W1219 10:32:26.479931128 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destro
y_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This
 constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
W1219 10:32:27.427000 2498183 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2498202 closing signal SIGTERM
E1219 10:32:27.591000 2498183 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 2498201) of binary: /environments/llama_factory_uat/bin/python3.11
Traceback (most recent call last):
  File "/environments/llama_factory_uat/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

my llama3_lora_predict.yaml

### model
model_name_or_path: /Llama-3.3-70B-Instruct-bnb-4bit
adapter_name_or_path: saves/llama3.3-70b/lora/sft
trust_remote_code: true

### method
stage: sft
do_predict: true
finetuning_type: lora

### dataset
eval_dataset: JB_translate_test
template: llama3
cutoff_len: 4096
#max_samples: 50
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/llama3.3-70b/lora/predict
overwrite_output_dir: true

### eval
per_device_eval_batch_size: 1
predict_with_generate: true
ddp_timeout: 180000000

Expected behavior

No response

Others

No response

The text was updated successfully, but these errors were encountered:

hiyouga · 2024-12-19T10:37:02Z

git pull origin main

AI-LMD · 2024-12-19T10:43:35Z

哥，还是不行，我也遇到了相同问题

Kizooo1 · 2024-12-19T10:52:52Z

git pull origin main

作者您好，我这边也遇到了同样的问题，按您的操作了也是不行，以前都没有这样的问题，今天突然出现，是否是版本更新原因？

AI-LMD · 2024-12-19T11:01:32Z

git pull origin main

作者您好，我这边也遇到了同样的问题，按您的操作了也是不行，以前都没有这样的问题，今天突然出现，是否是版本更新原因？

哥们，问题解决了吗？是transformer库不适配吗？

hiyouga · 2024-12-19T12:24:30Z

fixed

harrybattler · 2024-12-19T14:55:24Z

I can help you.
Contact with me.

paolovic · 2024-12-19T15:19:23Z

Hi @hiyouga @harrybattler ,
unfortunately not fixed.

This is my checked out commit:

LLaMA-Factory]$ git log --pretty=format:'%H' -n 1
ffbb4dbdb09ba799af1800c78b2e9d669bccd24b

llamafactory-cli env

llamafactory version: 0.9.2.dev0
Platform: Linux-4.18.0-553.27.1.el8_10.x86_64-x86_64-with-glibc2.28
Python version: 3.11.10
PyTorch version: 2.5.1+cu124 (GPU)
Transformers version: 4.46.1
Datasets version: 3.1.0
Accelerate version: 1.0.1
PEFT version: 0.12.0
TRL version: 0.9.6
GPU type: NVIDIA L40S-48C
Bitsandbytes version: 0.45.0

Traceback (most recent call last):
  File "/environments/llama_factory_uat/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py",
line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/packages/LLaMA-Factory/src/llamafactory/launcher.py FAILED

paolovic · 2024-12-19T16:01:01Z

could you reopen it, please? @hiyouga

AI-LMD · 2024-12-20T02:44:17Z

Hi @hiyouga @harrybattler , unfortunately not fixed.

This is my checked out commit:

LLaMA-Factory]$ git log --pretty=format:'%H' -n 1
ffbb4dbdb09ba799af1800c78b2e9d669bccd24b

llamafactory-cli env

llamafactory version: 0.9.2.dev0
Platform: Linux-4.18.0-553.27.1.el8_10.x86_64-x86_64-with-glibc2.28
Python version: 3.11.10
PyTorch version: 2.5.1+cu124 (GPU)
Transformers version: 4.46.1
Datasets version: 3.1.0
Accelerate version: 1.0.1
PEFT version: 0.12.0
TRL version: 0.9.6
GPU type: NVIDIA L40S-48C
Bitsandbytes version: 0.45.0

Traceback (most recent call last):
  File "/environments/llama_factory_uat/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py",
line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/packages/LLaMA-Factory/src/llamafactory/launcher.py FAILED

get pull origin main can resolve my question ,if you don't resolve it ,i think you should check your transformers version ,such as replace it 4.43.4

paolovic · 2024-12-20T07:46:35Z

Hi @hiyouga @harrybattler , unfortunately not fixed.
This is my checked out commit:

LLaMA-Factory]$ git log --pretty=format:'%H' -n 1
ffbb4dbdb09ba799af1800c78b2e9d669bccd24b

llamafactory-cli env

llamafactory version: 0.9.2.dev0
Platform: Linux-4.18.0-553.27.1.el8_10.x86_64-x86_64-with-glibc2.28
Python version: 3.11.10
PyTorch version: 2.5.1+cu124 (GPU)
Transformers version: 4.46.1
Datasets version: 3.1.0
Accelerate version: 1.0.1
PEFT version: 0.12.0
TRL version: 0.9.6
GPU type: NVIDIA L40S-48C
Bitsandbytes version: 0.45.0

Traceback (most recent call last):
  File "/environments/llama_factory_uat/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py",
line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/environments/llama_factory_uat/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/packages/LLaMA-Factory/src/llamafactory/launcher.py FAILED

get pull origin main can resolve my question ,if you don't resolve it ,i think you should check your transformers version ,such as replace it 4.43.4

I just downgraded transformers to 4.43.4 and it didn't help
I will create a new issue for more visibility

github-actions bot added the pending This problem is yet to be addressed label Dec 19, 2024

hiyouga added solved This problem has been already solved and removed pending This problem is yet to be addressed labels Dec 19, 2024

hiyouga closed this as completed Dec 19, 2024

hiyouga reopened this Dec 19, 2024

hiyouga mentioned this issue Dec 19, 2024

[generate] fix generate kwargs #6395

Merged

2 tasks

hiyouga closed this as completed in #6395 Dec 19, 2024

hiyouga closed this as completed in d4c1fda Dec 19, 2024

paolovic mentioned this issue Dec 20, 2024

ValueError: The following model_kwargs are not used by the model: ['skip_special_tokens'] #6403

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] #6391

ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] #6391

paolovic commented Dec 19, 2024

hiyouga commented Dec 19, 2024

AI-LMD commented Dec 19, 2024

Kizooo1 commented Dec 19, 2024

AI-LMD commented Dec 19, 2024

hiyouga commented Dec 19, 2024

harrybattler commented Dec 19, 2024

paolovic commented Dec 19, 2024 •

edited

Loading

paolovic commented Dec 19, 2024

AI-LMD commented Dec 20, 2024

paolovic commented Dec 20, 2024

ValueError: The following model_kwargs are not used by the model: ['skip_special_tokens'] #6391

ValueError: The following model_kwargs are not used by the model: ['skip_special_tokens'] #6391

Comments

paolovic commented Dec 19, 2024

Reminder

System Info

Reproduction

Expected behavior

Others

hiyouga commented Dec 19, 2024

AI-LMD commented Dec 19, 2024

Kizooo1 commented Dec 19, 2024

AI-LMD commented Dec 19, 2024

hiyouga commented Dec 19, 2024

harrybattler commented Dec 19, 2024

paolovic commented Dec 19, 2024 • edited Loading

paolovic commented Dec 19, 2024

AI-LMD commented Dec 20, 2024

paolovic commented Dec 20, 2024

ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] #6391

ValueError: The following `model_kwargs` are not used by the model: ['skip_special_tokens'] #6391

paolovic commented Dec 19, 2024 •

edited

Loading