The code is based on LLaVA-Rad original github repo.

The major modification to the source code is at ./llava/train/train_forTopicSeg.py and ./llava/eval/
We add code to enable the parallel input of both the original image and segmented image.

# To run the code
You only need to make modifications to the scripts under ./MyScripts.

Replace each todo in each script with the name of the real directory containing the data or the finetuned model.

After you get the model, make sure to add a prefix "llava" to the front of the parent folder of the model.


# Note
However, as both MIMIC-CXR and IU-Xray datasets exceed the size limitations of the appendix, we cannot upload the images or the segmented images for both datasets.

Also, for the topic-split reports, since MIMIC-CXR-JPG contains 1,161,753 reports, we also cannot upload it here.

However, we have uploaded the processed reports for the IU-Xray dataset.

We will make all the data publicly available, including the model checkpoints, the segmented images and the processed reports. 