Issues running Spark Notebook

[ronan_b6]

Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting.

But today, notebooks do not start the spark session, returning this type of error:

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System."

I see from the forum that others have experience similar issues, with severe delays in functionality.

Can anyone shed more light on this?

Thanks,

[dmedeiros]

I am also facing the same issue since yesterday both using the trial capacity and a reserved capacity - both from Canada Central

[kausthab]

Facing the same issue in Ontario, Canada

[dver344]

Having a similar issue. It's not timing out, since I don't do a lot of complex processes, but the spark pools usually takes 3 to 5 min to start (before the issue this was 10sec). This all started 24h ago for me (time of writing is 17/07/2024, 2PM Brussels)

[maxmerchant]

I, too, am encountering this issue since yesterday. We were able to run notebooks without any issues until last week and started seeing the error below. Tried several different notebooks in different workspaces and all give the same error.

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System"

We are using Fabric F64 instance. Region Canada Central.

Is anyone else having this issue as well?

[naveesupply]

Same issue, were you able to fix it? 

[maxmerchant]

No. Still facing this issue. The weird thing is that it Livy session starts every once in a while, with no consistency. So can't execute a pipeline with a notebook activity.

[naveesupply]

Ya I am just running a script using the notebook but its not connecting at all. Will have to wait to see if it works tomorrow. 

[Anonymous]

Hi @ronan_b6 ,

May I ask if you are still facing this issue?
You can try to restart the spark pool to resolve the issue and make sure that the spark is configured correctly and there are enough available resources in the spark pool.
If the issue is still not resolved, can you provide more details like in which region you are facing this issue and attach a screenshot of the issue.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[benfedit]

hi Ada we have the same issue here in North Europe all our notebooks in Fabric are affected.

[Sarien]

Still facing the same issue as well. Tried changing the environment version as well which did not help.

Not sure how I should restart the pool as you are saying.

[JFTxJ]

Can you please clarify how to "restart the spark pool"?

Thank you

[naveesupply]

Hi Ada, 

How would I restart the spark pool? I am facing the same issue on my trial, I am also based out of Canada