ostris/flux-dev-lora-trainer api call failed with 500 but training job actually went through, started and finished.

[fyepi]

I am using the JS client to submit lora training job. It was all good before yesterday.

The end point returns 500 with body: {"detail":"An unexpected error ocurred"}

But in dashboard the training task was started and finished successfully.

The code is exactly like what it is at the model page:

const training = await replicate.trainings.create(
  "ostris",
  "flux-dev-lora-trainer",
  "26dce37af90b9d997eeb970d92e47de3064d46c300504ae376c75bef6a9022d2",
  {
    // You need to create a model on Replicate that will be the destination for the trained version.
    destination: "xxx/xxxxx",
    input: {
      steps: 1000,
      lora_rank: 16,
      optimizer: "adamw8bit",
      batch_size: 1,
      resolution: "512,768,1024",
      autocaption: true,
      input_images: "https://xxxxx",
      trigger_word: "TOK",
      learning_rate: 0.0004,
      wandb_project: "flux_train_replicate",
      wandb_save_interval: 100,
      caption_dropout_rate: 0.05,
      cache_latents_to_disk: false,
      wandb_sample_interval: 100,
      gradient_checkpointing: false
    }
  }
);

I have submitted a support request from replicate.com but received no response at all.

It seems replicate's service degenerated a lot since it was sold.